Principles in Patterns (PiP) : Evaluation of Impact on Business Processes

The innovation and development work conducted under the auspices of the Principles in Patterns (PiP) project is intended to explore and develop new technology-supported approaches to curriculum design, approval and review. An integral component of this innovation is the use of business process analysis and process change techniques - and their instantiation within the C-CAP system (Class and Course Approval Pilot) - in order to improve the efficacy of curriculum approval processes. Improvements to approval process responsiveness and overall process efficacy can assist institutions in better reviewing or updating curriculum designs to enhance pedagogy. Such improvements also assume a greater significance in a globalised HE environment, in which institutions must adapt or create curricula quickly in order to better reflect rapidly changing academic contexts, as well as better responding to the demands of employment marketplaces and the expectations of professional bodies. This is increasingly an issue for disciplines within the sciences and engineering, where new skills or knowledge need to be rapidly embedded in curricula as a response to emerging technological or environmental developments. All of the aforementioned must also be achieved while simultaneously maintaining high standards of academic quality, thus adding a further layer of complexity to the way in which HE institutions engage in "responsive curriculum design" and approval. This strand of the PiP evaluation therefore entails an analysis of the business process techniques used by PiP, their efficacy, and the impact of process changes on the curriculum approval process, as instantiated by C-CAP. More generally the evaluation is a contribution towards a wider understanding of technology-supported process improvement initiatives within curriculum approval and their potential to render such processes more transparent, efficient and effective. Partly owing to limitations in the data required to facilitate comparative analyses, this evaluation adopts a mixed approach, making use of qualitative and quantitative methods as well as theoretical techniques. These approaches combined enable a comparative evaluation of the curriculum approval process under the "new state" (i.e. using C-CAP) and under the "previous state". This report summarises the methodology used to enable comparative evaluation and presents an analysis and discussion of the results. As the report will explain, the impact of C-CAP and its ability to support improvements in process and document management has resulted in the resolution of numerous process failings. C-CAP has also demonstrated potential for improvements in approval process cycle time, process reliability, process visibility, process automation, process parallelism and a reduction in transition delays within the approval process, thus contributing to considerable process efficiencies; although it is acknowledged that enhancements and redesign may be required to take advantage of C-CAP's potential. Other aspects pertaining to C-CAP's impact on process change, improvements to document management and the curation of curriculum designs will also be discussed

Z39.50 broadcast searching and Z-server response times: perspectives from CC-interop

This paper begins by briefly outlining the evolution of Z39.50 and the current trends, including the work of the JISC CC-interop project. The research crux of the paper focuses on an investigation conducted with respect to testing Z39.50 server (Z-server) response times in a broadcast (parallel) searching environment. Customised software was configured to broadcast a search to all test Z-servers once an hour, for eleven weeks. The results were logged for analysis. Most Z-servers responded rapidly. 'Network congestion' and local OPAC usage were not found to significantly influence Z-server performance. Response time issues encountered by implementers may be the result of non-response by the Z-server and how Z-client software deals with this. The influence of 'quick and dirty' Z39.50 implementations is also identified as a potential cause of slow broadcast searching. The paper indicates various areas for further research, including setting shorter time-outs and greater end-user behavioural research to ascertain user requirements in this area. The influence more complex searches, such as Boolean, have on response times and suboptimal Z39.50 implementations are also emphasised for further study. This paper informs the LIS research community and has practical implications for those establishing Z39.50 based distributed systems, as well as those in the Web Services community. The paper challenges popular LIS opinion that Z39.50 is inherently sluggish and thus unsuitable for the demands of the modern user

Principles in Patterns (PiP) : Project Evaluation Synthesis

Evaluation activity found the technology-supported approach to curriculum design and approval developed by PiP to demonstrate high levels of user acceptance, promote improvements to the quality of curriculum designs, render more transparent and efficient aspects of the curriculum approval and quality monitoring process, demonstrate process efficacy and resolve a number of chronic information management difficulties which pervaded the previous state. The creation of a central repository of curriculum designs as the basis for their management as "knowledge assets", thus facilitating re-use and sharing of designs and exposure of tacit curriculum design practice, was also found to be highly advantageous. However, further process improvements remain possible and evidence of system resistance was found in some stakeholder groups. Recommendations arising from the findings and conclusions include the need to improve data collection surrounding the curriculum approval process so that the process and human impact of C-CAP can be monitored and observed. Strategies for improving C-CAP acceptance among the "late majority", the need for C-CAP best practice guidance, and suggested protocols on the knowledge management of curriculum designs are proposed. Opportunities for further process improvements in institutional curriculum approval, including a re-engineering of post-faculty approval processes, are also recommended

Principles in Patterns (PiP) : User Acceptance Testing of Course and Class Approval Online Pilot (C-CAP)

The PiP Evaluation Plan documents four distinct evaluative strands, the first of which entails an evaluation of the PiP system pilot (WP7:37 – Systems & tool evaluation). Phase 1 of this evaluative strand focused on the heuristic evaluation of the PiP Course and Class Approval Online Pilot system (C-CAP) and was completed in December 2011. Phase 2 of the evaluation is broadly concerned with "user acceptance testing". This entails exploring the extent to which C-CAP functionality meets users' expectations within specific curriculum design tasks, as well as eliciting data on C-CAP's overall usability and its ability to support academics in improving the quality of curricula. The general evaluative approach adopted therefore employs a combination of standard Human-Computer Interaction (HCI) approaches and specially designed data collection instruments, including protocol analysis, stimulated recall and pre- and post-test questionnaire instruments. This brief report summarises the methodology deployed, presents the results of the evaluation and discusses their implications for the further development of C-CAP

Principles in Patterns (PiP) : Piloting of C-CAP - Evaluation of Impact and Implications for System and Process Development

The Principles in Patterns (PiP) project is leading a programme of innovation and development work intended to explore and develop new technology-supported approaches to curriculum design, approval and review. It is anticipated that such technology-supported approaches can improve the efficacy of curriculum approval processes at higher education (HE) institutions, thereby improving curriculum responsiveness and enabling improved and rapid review mechanisms which may produce enhancements to pedagogy. Curriculum design in HE is a key "teachable moment" and often remains one of the few occasions when academics will plan and structure their intended teaching. Technology-supported curriculum design therefore presents an opportunity for improving academic quality, pedagogy and learning impact. Approaches that are innovative in their use of technology offer the promise of an interactive curriculum design process within which the designer is offered system assistance to better adhere to pedagogical best practice, is exposed to novel and high impact learning designs from which to draw inspiration, and benefits from system support to detect common design issues, many of which can delay curriculum approval and distract academic quality teams from monitoring substantive academic issues. This strand of the PiP evaluation (WP7:38) attempts to understand the impact of the PiP Class and Course Approval Pilot (C-CAP) system within specific stakeholder groups and seeks to understand the extent to which C-CAP is considered to support process improvements. As process improvements and changes were studied in a largely quantitative capacity during a previous but related evaluative strand, this strand includes the gathering of additional qualitative data to better understand and verify the business process improvements and change effected by C-CAP. This report therefore summarises the outcome of C-CAP piloting within a University faculty, presents the methodology used for evaluation, and the associated analysis and discussion. More generally this report constitutes an additional evaluative contribution towards a wider understanding of technology-supported approaches to curriculum design and approval in HE institutions and their potential in improving process transparency, efficiency and effectiveness

Principles in Patterns (PiP) : Heuristic Evaluation of Course and Class Approval Online Pilot (C-CAP)

The PiP Evaluation Plan documents four distinct evaluative strands, the first of which entails an evaluation of the PiP system pilot (WP7:37). Phase 1 of this evaluative strand focuses on the heuristic evaluation of the PiP Course and Class Approval Online Pilot system (C-CAP). Heuristic evaluation is an established usability inspection and testing technique and is most commonly deployed in Human-Computer Interaction (HCI) research, e.g. to test user interface designs, technology systems testing, etc. The success of heuristic evaluation in detecting 'major' and 'minor' usability problems is well documented, but its principal limitation is its inability to capture data on all possible usability problems. For this reason heuristic evaluation is often used as a precursor to user testing, e.g. so that user testing focuses on deeper system issues rather than on those that can easily be debugged. Heuristic evaluation nevertheless remains an important usability inspection technique and research continues to demonstrate its success in detecting usability problems which would otherwise evade detection in user testing sessions. For this reason experts maintain that heuristic evaluation should be used to complement user testing. This is reflected in the PiP Evaluation Plan, which proposes protocol analysis, stimulated recall and pre- and post-test questionnaire instruments to comprise user testing (see WP7:37 phases 2, 3 and 4 of PiP Evaluation Plan). This brief report summarises the methodology deployed, presents the results of the heuristic evaluation and proposes solutions or recommendations to address the heuristic violations that were found to exist in the C-CAP system. It is anticipated that some solutions will be implemented within the lifetime of the project. This is consistent with the incremental systems design methodology that PiP has adopted

Towards improved performance and interoperability in distributed and physical union catalogues

Purpose of this paper: This paper details research undertaken to determine the key differences in the performance of certain centralised (physical) and distributed (virtual) bibliographic catalogue services, and to suggest strategies for improving interoperability and performance in, and between, physical and virtual models. Design/methodology/approach: Methodically defined searches of a centralised catalogue service and selected distributed catalogues were conducted using the Z39.50 information retrieval protocol, allowing search types to be semantically defined. The methodology also entailed the use of two workshops comprising systems librarians and cataloguers to inform suggested strategies for improving performance and interoperability within both environments. Findings: Technical interoperability was permitted easily between centralised and distributed models, however the various individual configurations permitted only limited semantic interoperability. Significant prescription in cataloguing and indexing guidelines, greater participation in the Program for Collaborative Cataloging (PCC), consideration of future 'FRBR' migration, and greater disclosure to end users are some of the suggested strategies to improve performance and semantic interoperability. Practical implications: This paper informs the LIS research community and union catalogue administrators, but also has numerous practical implications for those establishing distributed systems based on Z39.50 and SRW, as well as those establishing centralised systems. What is original/value of the paper?: The paper moves the discussion of Z39.50 based systems away from anecdotal evidence and provides recommendations based on testing and is intimately informed by the UK cataloguing and systems librarian community

Digital libraries and information literacy issues within virtual learning environments : an e-learning impasse?

The DIDET digital library and VLE approach places much of the responsibility for managing the digital library work flow into the hands of students, as well as academics and librarians. Student responsibilities include the application of metadata, as well as conventional information literacy competencies such as ascertaining information resource provenance, investigating intellectual property rights and/or digital rights management implications, before depositing digital resources within the library. This has obviously laid bare numerous research issues relating to future digital library and VLE design, student information literacy, the use of ICT in education and design, and related pedagogical issues, all of which are worthy of further investigation within the UK HE community and will be elucidated in this paper. More importantly, this paper will argue that such a model signifies a definite impasse in the evolution of e-learning models and questions the degree to which current information literacy models are effective in specific e-learning contexts. The paper will conclude by further recognising that greater student information literacy skills are necessary to unlock the potential of such radical approaches to e-learning and digital library creation

Collaborative tagging as a knowledge organisation and resource discovery tool

The purpose of the paper is to provide an overview of the collaborative tagging phenomenon and explore some of the reasons for its emergence. Design/methodology/approach - The paper reviews the related literature and discusses some of the problems associated with, and the potential of, collaborative tagging approaches for knowledge organisation and general resource discovery. A definition of controlled vocabularies is proposed and used to assess the efficacy of collaborative tagging. An exposition of the collaborative tagging model is provided and a review of the major contributions to the tagging literature is presented. Findings - There are numerous difficulties with collaborative tagging systems (e.g. low precision, lack of collocation, etc.) that originate from the absence of properties that characterise controlled vocabularies. However, such systems can not be dismissed. Librarians and information professionals have lessons to learn from the interactive and social aspects exemplified by collaborative tagging systems, as well as their success in engaging users with information management. The future co-existence of controlled vocabularies and collaborative tagging is predicted, with each appropriate for use within distinct information contexts: formal and informal. Research limitations/implications - Librarians and information professional researchers should be playing a leading role in research aimed at assessing the efficacy of collaborative tagging in relation to information storage, organisation, and retrieval, and to influence the future development of collaborative tagging systems. Practical implications - The paper indicates clear areas where digital libraries and repositories could innovate in order to better engage users with information. Originality/value - At time of writing there were no literature reviews summarising the main contributions to the collaborative tagging research or debate

Principles in Patterns (PiP) : Institutional Approaches to Curriculum Design Institutional Story

The principal outputs of the PiP Project surround the Course and Class Approval (C-CAP) system. This web-based system built on Microsoft SharePoint addresses and resolves many of the issues identified by the project. Generally well received by both academic and support staff, the system provides personalised views, adaptive forms and contextualised support for all phases of the approval process. Although the system deliberately encapsulates and facilitates existing approval processes thus achieving buy-in, it is already achieving significant improvements over the previous processes, not only in reducing the administrative overheads but also in supporting curriculum design and academic quality. The system is now embedded across three faculties and is now considered by the University of Strathclyde to be a "core institutional service". Alongside the C-CAP system the PiP Project also cultivated a suite of approaches: an incremental systems development methodology; a structured and replicable evaluation approach, and; Strathclyde's Lean Approach to Efficiencies in Education Kit (SLEEK) business process improvement methodology Each is based on recognised formal techniques, providing the basis for a rigorous approach. This is contextualised within and adapted to the HE institutional context thus building the foundation not only for the project but ultimately for institution wide process improvement. This "institutional story" report summarises the principal outcomes of the Project